Comparing interactive computer services-Theoretical, technical and economic feasibility

The comparison of interactive computer services is a frequent, important and essential activity for those involved in selecting from among alternative remote access services available through a computer network. The comparison and selection process can be very complicated, relying on both nonmeasurable and measurable comparison criteria. Nonmeasurable criteria, such as the availability of a particular compiler, are typically easy and inexpensive to evaluate. In contrast, for those criteria which are measurable, comparison calls for an expensive process of collecting and analyzing relevant performance measurements from the netwoik computer services under consideration. (The term "network computer services" is used to refer to interactive computer services available via a computer network). Further, if measurement is not carefully planned to accommodate certain inherent theoretical, technical and economic problems, the measurements which result may be misleading or invalid. A legitimate question arises as to whether comparison measurements should be done at all. This paper addresses the feasibility of including measurement phases in the process of comparing and selecting interactive network computer services. Two essential problems arise when measuring interactive services. The first is developing a test workload or "scenario" that is truly representative of a reai workioad. "'Representativeness" is a critical property since measurement experiments are designed so that the performance of an interactive service while processing the test workload is assumed to be close to what its performance would be while processing the real workload. This representativeness problem is not addressed here, but will be discussed in the accompanying paper. The second probiem in measuring interactive services arises while collecting and analyzing data about the performance of a test workload. Three aspects of the data collection and analysis problem must be considered